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PROVISIONAL PATENT APPLICATION OF 
Shoichi Endo, Jeffrey Eid, Dean Chang, and Erik Shahoian 

for 

, HAPTIC MESSAGING IN HANDHELD DEVICES 



FIELD OF THE INVENTION 
[0001] This invention relates generally to haptic-feedback systems. More 

specifically, embodiments of the present invention relate to using customized hapti< 
effects in a variety of contexts to convey information to users of handheld 
communication devices. 



BACKGROUND 

[0002] As handheld communication devices become part of everyday life, device 
manufactures and service providers strive to enhance the versatility and performance of 
such devices. 

[0003] ■ Handheld communication devices in the art (e.g., mobile phones, pagers, 
personal digital assistants (PDAs), etc.) typically use auditory and visual cues to alert a 
user when incoming messages, such as voice calls and emails, are received. Such 
auditory and visual alerts, however, have the disadvantages of being distracting in some 
situations (e.g., during driving), or annoying in others (e.g., during a meeting or a 
concert). Although vibratory alerts are also made available in some communication 
devices such as cellular phones, the vibratory effects that can be generated act in a "one- 
size-fitting-air fashion for a given device, as opposed to being customized according to 
applications, thus conveying very little information about the source of the alert. A need 
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exists therefore in the art, for delivering information to users of handheld communication 
devices in a manner that overcomes the aforementioned shortcomings. There would be a 
further advance in the art to provide a new sensory modality in handheld devices. 

SUMMARY 

[0004J Embodiments of the invention relate to methods and systems for providing 

« 

"haptic messaging" to users of handheld communication devices in a customized (or 
personalized) manner. 

10005] In one embodiment, a method of haptic messaging comprises: detennimng the 
source of a call event; selecting a predetermined signal based on the determination, the 
predetermined signal configured to correlate with a haptic effect; and outputting the 
haptic effect to a handheld device. 

10006] The method also includes supplying a collection of haptic effects, each 

correlating with a predetermined signal; and mapping each call event to one of the haptic 

effects (e.g., by way of its source). The one-to-one mappings between various call events 

of interest and corresponding haptic effects can be further compiled into a "haptic lookup 
table." 



BRIEF DESCRIPTION OF THE FIGURES 
[0007] These and other features, aspects, and advantages of the present in vention are 

ft * 

better understood when the following Detailed Description is read with reference to the 
accompanying drawings, wherein: 



FIG. 1 shows a flowchart depicting a method of haptic messaging i 
handheld device, according to one embodiment of the present invention; 
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FIG. 2 depicts a haptic handheld device according to one embodiment of the 
present invention; 

FIG. 3 is a flowchart illustrating a process of providing a remote haptic effect 
in one embodiment of the present invention; and 

FIG. 4 is a flowchart illustrating a process for providing haptic effects to a 
remote control in one embodiment of the present invention. 

DETAILED DESCRIPTION 

* 

[0008] FIG. 1 shows a flowchart 100 depicting a method of haptic messaging in a 
handheld device, according to an embodiment of the invention. It will be appreciated that 
the embodiment of FIG. 1 is provided by way of example to illustrate the principles of 
the invention, and should not be construed as limiting the scope of the invention in any 
manner. One skilled in the art would also recognize that various changes and 
modifications can be made herein, without departing from the principles and scope of the 
invention. 



[0009] The flowchart 1 00 of FIG. 1 comprises determining a source of a call event, 
recited in step 1 10; selecting a predetermined signal based on the determination, the 
predetermined signal configured to correlate with a haptic effect, as recited in step 120; 
and outputting the haptic effect to a handheld device, as recited in step 130. 

[0010] In the embodiment of FIG. 1 , the term "call event" comprises any message 
that conveys information to the user of a handheld device. Examples of call event 
include an incoming message, such as a voice call, an email, a text message (e.g., an 
"instant message" as known in the art), or a virtual touch indicator/signal. Examples of 
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call event can also include a self-generated message, such as a reminder for a pre- 
scheduled activity (e.g., an appointment or a show). The "source" relates to a 
characteristic that distinctly identifies (or characterizes) a call event, such as the caller's 
phone number, the sender's email address, or the type of a pre-scheduled activity. The 
term "selecting" includes, without limitation, looking up a predetermined mapping 
between the call event at play and a corresponding haptic effect, and selecting/generating 
a predetermined signal that is configured to render the desired haptic effect (e.g., upon 
being applied to an actuator). 

[00111 The flowchart of FIG. 1 includes supplying a collection of haptic effects, each 
correlating with a predetermined signal, as recited in step 140. Underlying this collection 
of haptic effects can be a software code (e.g., stored in a memory on the handheld device) 
that is devised to generate the corresponding predetermined signals (e.g., each 
characterized by a distinct waveform) rendering these haptic effects. A user can map 
each call event to one of the stored haptic effects (e.g., by way of its source), as recited in 
step 150 of FIG. 1. And the one-to-one mappings made between various call events of 
interest and the corresponding haptic effects can be compiled into a "haptic lookup table" 
(e.g., stored in a memory on the handheld device), as recited in step 160. The "selection" 
recited in step 120 can be made based upon this haptic lookup table, for instance. 

[0012] As such, each call event is accompanied by a distinct haptic effect, conveying 
to the user customized information such as "who is calling," "what is happening," and so 
on. The user can also be allowed to update the "haptic lookup table," e.g., to include new 
call events, and/or to modify the mappings between the existing call events and 

4 

corresponding haptic effects. 
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[0013] In the above, the term "handheld communication device" (or "handheld 
device") includes a mobile phone such as a cellular phone or a satellite phone (e.g., a 
blackberry" as known in the art), a PDA, a pager, a handheld (or portable) computer, a 
gamepad, or other handheld devices known in the art that are equipped with 
communication (or networking) capabilities. 

[0014] In one embodiment, the aforementioned haptic effects can serve as "haptic 
ringers" (e.g., "counterparts" to auditory ring tones), to inform the user of events such as 

* 

incoming voice calls and emails. Such haptic ringers can be customized (or personalized) 
according to caller phone numbers and/or sender's email addresses, so as to allow the 
user to identify the source of an incoming message. As a way of example, a haptic ringer 
associated with a call from a loved one (e.g., the user's mother) may comprise low- 
amplitude and higji frequency vibrations that impart "gentle" sensations to the user. In 
contrast, a haptic ringer for a "91 1" call may comprise "jolt-like" pulses that impart 
founding" sensations to the user. Haptic ringers can also be customized to inform the 
user of self-generated events such as pre-scheduled activities, e.g., by way of the activity 
type (whether it be a business meeting or a favorite TV program, for instance). 

[0015] In contrast with conventional auditory ring tones, the aforementioned haptic 
ringers are more desirable in an environment where extraneous auditory signals are 
prohibited (e.g., during a meeting or a concert), and/or where it is difficult to distinguish 
auditory signals (e.g., in a loud environment such as an airport). The haptic ringers are 
also more suitable in a situation such as driving, so that the user of a handheld device can 
keep eyes on the road without having to look at the device. Moreover, such haptic 
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ringers convey customized information to the user, so that the user is aware of "who is 
calling," "what is happening," and so on. 

[001 6] Embodiments of the present invention include the following: 
[0017] Examples 

[0018J In one embodiment, the individual's telephone (cellular or otherwise) includes 
e-mail capability (both receive and send). The telephone also includes hardware and 
software that provides a third distinctive vibration - different from the first two 
mentioned above - when an e-mail is received from the e-mail address of the individual's 
brother. It further includes hardware and software that provides a fourth distinctive 
vibration - different from the first three mentioned - when an e-mail is received from 
the e-mail address of the individual's top client if the e-mail contains the "smiley-face" 
emoticon : ). The fourth distinctive vibration is a short haptic effect of higher magnitude, 
meant to emulate a "high five." The individual programs the telephone by inputting the 
respective e-mail addresses and selecting a vibration to use with each. When calls are 
subsequently received, the telephone provides the effect. 

[0019] In another embodiment, the individual is in a movie theater, watching a 
movie, and has the telephone in the individual's pocket It is set not to make any noise, 
since the user is in the theater. While the individual is watching the n\ovie, the 
individual's telephone vibrates with the second vibration. The individual chooses to 
ignore the call, because the individual does not wish to speak with anyone at the 
individual's place of employment. Later, the individual's telephone vibrates with the first 
vibration. The individual's wishes to speak with the individual's spouse, to make plans to 

6 
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meet later, and so the individual answers the phone and quickly exits the theater to talk 
with the individual's spouse. 

[0020] The individual's telephone includes a personal schedule calendar application. 

After speaking with the individual's spouse, the individual enters an entry in the calendar 

at the 7:00 pm time mark "Meet Cathy" (his spouse). The individual also chooses a fifth 

distinctive vibration — different from the first four mentioned - to associate with the 

» 

calendar entry. The telephone is programmed to provide the fifth distinctive vibration 
fifteen minutes before the time entry of the calendar (i.e., at 6:45). 

[0021] FIG. 2 depicts a haptic handheld device 200 according to an embodiment of 
the invention, illustrating by way of example how the embodiment of FIG. 1 can be 
implemented. It will be appreciated that various elements are shown in schematic form 
for illustrative purposes and are not drawn to scale. It will also be appreciated that there 
are many alternative ways of practicing the present invention. Accordingly, various 
changes and modifications may be made herein, without departing from the principles 
and scope of the invention. 

[0022] By way of example, haptic device 200 may include a device body including a 
housing 210 and a user-interface 212; a processor 220; at least one actuator 230; and a 
memory 240 storing a software code to be executed by the processor 220. For purpose of 
illustration, the processor 220, the actuator 230, and the memory 240 are shown to be 
enclosed within and coupled to the device body. Such an illustration, however, should 
not be construed as limiting the scope of the invention in any manner. In alternative 
embodiments, for instance, the actuator may be coupled to the outside of the housing 210, 
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or embedded in the housing 210 via a suitable mechanism; Further, the user-interface 
210 may contain a key pad (containing one or more keys), a touch pad or touch screen, a 
scroll wheel, a direction pad, a miniature joystick, and/or other user-interface means 
known in the art. 

[0023] In the embodiment of FIG. 2, the executable software code stored in the 
memory 240 is configured to cause the processor 220 to perform tasks upon executing the 
software. More specifically, the executable software code causes the processor 220 to 
determine the source of a call event 250 and based on the determination, to select a 
predetermined signal that correlates with a haptic effect The executable software code 
further causes the processor 220 to command the selected predetermined signal to the 
actuator 230, such that the haptic effect is rendered and output to the device body. 

[0024] The memory 240 may further store code that is devised to generate a plurality 
of predetermined signals, each correlating with a haptic effect, such that a collection of 
haptic effects is made available on the handheld device. The memory 240 may also store 
code that enables a user to map a call event of interest to one of such haptic effects, In 
addition, a user may be allowed to compile the one-to-one mappings between various call 
events of interest and corresponding haptic effects into a "haptic lookup table,** which can 
also be stored in the memory 240, for instance. 

[0025] The selection of a predetermined signal described abQve may be carried out by 
way of looking up the aforementioned "haptic lookup table," and selecting/generating the 
corresponding predetermined signal that renders the desired haptic effect, upon being 
applied to the actuator 230. 
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[0026] Moreover, the executable software code may contain provisions for dealing 
with a call event whose source is unknown. For instance, a specific haptic effect can be 
designated for any unknown call event (such as a voice call or an email), so as to alert the 
user that the incoming message is from an un-identifiable caller (or sender). 

[0027] In the above, the actuator 230 may be a pager motor, an eccentric rotating 
motor, a harmonic eccentric rotating motor, a voice coil, a piezoelectric actuator, an 
electro-active polymer actuator, or other types of active/passive actuators suitable for 
generating haptic (e.g., tactile and kinesthetic) effects. U.S. Patent Nos. 6,429,846 and 
6,424,333 disclose further details relating to some of these actuators, both of which are 
incorporated in full herein by reference. It will be appreciated that in some instances, one 
or more actuators may be implemented in a handheld device, configured to deliver 
appropriate haptic effects. It will be further appreciated that various control schemes can 
be devised accordingly for controlling the actuator(s) in a manner that best achieves the 
desired haptic effects. 

[0028] In one embodiment, the actuator 230 may be coupled to the housing 210, 
thereby imparting the haptic effects thus generated to the device body. Haptic ringers (or 
alerts) described above may be delivered in this manner, for instance. In another 
embodiment, the actuator 230 may be coupled to the user-interface 212 of the device 
body. For instance, an active actuator can be coupled to a scroll wheel (or a joystick) on 
the user-interface 212 to deliver a virtual touch (e.g., handshake) described above, as a 
user moves the scroll wheel (or joystick) back and forth. One or more actuators can also 
be coupled to a keypad (or a touch pad) on the user- interface 212 to convey a "high-five" 
to a user, in another instance. 

9 
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[0029J The collection of haptic effects stored in a handheld device described above 
may include one or more for delivering remote haptic effects, including various virtual 
touch effects (such as "handshake" and "high-five"). FIG. 3 is a flowchart illustrating a 
process of providing a remote haptic effect in one embodiment of the present invention. 
In the embodiment shown, a call event in the form of a "virtual touch indicator" may be 
first transmitted to a handheld device (e.g., the haptic device 200 of FIG. 2), to inform the 
user that a remote user wishes to send a virtual touch such as a "handshake." The device 
(200) receives the call 302. A distinctive haptic ring, as described in relation to FIG. 1, 
may accompany the call, identifying the caller and the nature of the call. The device then 
answers the call, performing any necessary instantiation routines to enable the 
communication 304. The device (200) receives a signal in the communication mat 
contains the code or waveform for the haptic effect 306. The device (200) provides the 
haptic effect 308. 

[0030] For example, the haptic effect may include a "handshake." The call includes a 
distinctive ring, alerting the user to the fact that an effect will be forthcoming. 
Accordingly, the user of the device (200) maintains contact with an appropriate 
"touching" means such as a scroll wheel described above, and experiences the 
"handshake" in an interactive (or real-time) fashion. 

I0031J Referring again to FIG. 3, the device (200) determines if additional effects are 
contained or forthcoming in the communication 310. If so, the device (200) receives the 
signal 306 and provides the effect 308. If not, the device (200) ends the call. It will be 
appreciated that in another embodiment, a "virtual touch signal" may be transmitted to 
the handheld device (e.g., following the virtual touch indicator), and renders the 

10 
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corresponding virtual touch on the user. From the teaching of the invention, one skilled 
in the art can also devise other schemes for delivering virtual touch on users of handheld 
devices. 

*&. - 

[0032] In another embodiment, the haptic effects of the invention can be used to 
provide a new dimension of sensing to users of handheld devices, such as delivering 
virtual touch in an interactive (or real-time) manner. For instance, kinesthetic effects can 



be devised to convey a virtual "handshake," a virtual "high-five," or a virtual "massage" 
between two users of two handheld devices in a real-time fashion. The ensuing 
description provides further detail in this regard. 

[0033] A haptic-enabled handheld device of the invention (e.g., the embodiment of 
FIG. 2) may be further used as a "two-way" remote control, e.g, for controlling a remote 
system such as a Television set or a multimedia system. In this scenario, the "call 
events" as referred to above may be related to program channels shown on the remote 
system, each identified by a channel number (which may be used as the "source"), for 
instance. The corresponding haptic effects may be customized on a per-channel basis. 
Such haptic effects can serve to inform a user as to which channel is on, as a user is 
channel-surfing by way of this haptic remote control, so that the user need not to look up 
the display screen. 

[0034] • FIG. 4 is a flowchart illustrating a process for providing haptic effects to a 
remote control in one embodiment of the present invention. In the embodiment shown, 
the remote control sends a command signal to the television or other device 402. As with 

* 

conventional remotes, the signal may or may not reach the device. Therefore, the remote 
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control then determines whether a feedback signal has been received 404. If not, the 
process ends. If the remote receives a feedback signal, the remote utilizes the feedback 
signal to determine the appropriate haptic effect to provide 406. 

[0035] In one embodiment, the television or other device provides information to the 
remote regarding the state of the display. In such an embodiment, the remote uses the 
information provided to determine the effect to provide. In another embodiment, the 
television determines the appropriate effect to provide and sends an identifier to the 
remote in the feedback signal. Once the remote has determined the appropriate haptic 
effect to provide, the remote provides the effect 408. 

[0036] An individual has a cellular telephone according to the present invention 

in the individual's pocket. The telephone includes programming that provides a first 
distinctive haptic effect (in this case, a particular vibration) when a call is received from 
the telephone number of the individual's spouse. The telephone also includes 
programming that provides a second vibration — different and distinct from the first 
vibration — when a call is received from the telephone number of the user's place of 
employment. The individual previous programmed the telephone by inputting the 
respective telephone numbers and selecting a vibration to use with each. 

[0037] One embodiment of the present invention implements a method, comprising 
determining a source of a call event, selecting a predetermined signal based on the 
determination, outputting to a handheld device a haptic effect correlating to the 
predetermined signal. The call event may comprise, for example, a call from another 
party or a self-generated message. A plurality of haptic effects may be stored in a lookup 
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■ 

table, wherein each haptic effect is correlated to one or more predetermined signals. The 
effects may include a haptic ringer, a virtual touch, or other desired effect 

[0038] A device according to one embodiment of the present invention comprises a 
body, a user interface coupled to the body, a processor enclosed within the body, an 
actuator coupled to the body and configured to output haptic effects to the body, and 
memory enclosed within the body. The device may comprise, for example, a cellular 
phone, a satellite phone, a pager, a personal digital assistant, an MP3 player, a game 
console controller, a personal gaming device, or a personal computer. The user interface 
of the device comprises a scroll wheel, a plurality of buttons, or other suitable interface 
elements. 

[0039] The foregoing description of the preferred embodiments of the invention has 
been presented only for the purpose of illustration and description and is not intended to 
be exhaustive or to limit the invention to the precise forms disclosed. Numerous 
modifications and adaptations thereof will be apparent to those skilled in the art without 
departing from the spirit and scope of the present invention. 
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